Collaboration system including a spatial event map

ABSTRACT

A spatial event map system including server-side data processor that maintains a spatial event map which locates events in a workspace. The spatial event map includes a log of events, entries in the log having a location of a graphical target of the event in the workspace and a time. The system includes logic to send messages including an event, a location of a graphical target of the event in the workspace and a time, to client-side network nodes; and to receive messages identifying events that create or modify a graphical target, and to add corresponding entries to the log of events. The events can include history events that are sent to the other client-side network nodes, and added to the log for the corresponding history events, and ephemeral events that are sent to other client-side network nodes without adding corresponding entries in the log.

CROSS-REFERENCE TO OTHER APPLICATIONS

The present application claims benefit of U.S. Provisional Applicationfor Patent No. 61/832,106 filed on 6 Jun. 2013; and

is a continuation-in-part of U.S. application Ser. No. 13/759,017 filedon 4 Feb. 2013, entitled COLLABORATION SYSTEM WITH WHITEBOARD ACCESS TOGLOBAL COLLABORATION DATA, now U.S. Pat. No. 9,479,548.

BACKGROUND

The invention relates to apparatuses, methods, and systems for digitalcollaboration, and more particularly to digital display systems whichfacilitate multiple simultaneous users having access to global workspacedata.

Digital displays are often used for interactive presentations and otherpurposes in a manner analogous to whiteboards. Some displays arenetworked and can be used for collaboration, so that modifications madeto the display image on one display are replicated on another display.Large scale displays offer the opportunity for more than one user topresent or annotate simultaneously on the same surface. However,problems can occur in the coordination of the multiple users, and insome circumstances their use of a single display can restrict theirflexibility of expression.

Also, digital displays can comprise large display screens or arrays ofscreens in a single room, which are configured to provide a largeinteraction surface. Thus, it is anticipated that the large digitaldisplays may be shared by many users at different times for differentcollaborations. Where the workspace data for collaboration isconfidential with access limited to authorized users, but the digitaldisplays at which the users interact are distributed to many sites andnot necessarily under exclusive control of a single user, a problemarises with the security of access to a collaboration.

In addition, the distributed nature of the system leads to thepossibility of multiple users in different places who interact with, andcan change, the same workspace data at the same time, and at times whenno other user is observing the workspace data. This creates a problemwith concurrency in the multiple locations, and with sharing informationabout a current state of the workspace data.

Therefore, it would be desirable to find ways to allow multiple users toshare workspace data in a distributed network of displays, in such a waythat each user has maximum freedom to express his or her ideas with realtime exchange of ideas, while providing security adequate to protect theconfidential nature of the collaboration. An opportunity thereforearises to create robust solutions to the problem. Better ideas,collaboration and results may be achieved.

SUMMARY

A collaboration system that implements a spatial event map is describedthat can have many distributed digital displays which are used todisplay images based on the spatial event map. A spatial event maps canalso be deployed in systems having a single display in a singlelocation.

A system that supports an essentially unlimited amount of 2D and 3Dworking space for each session that is accessible across multipledevices and locations is disclosed.

A collaboration system is described based on a spatial event map, whichincludes entries that locate events in a workspace. The spatial eventmap can include a log of events, where entries in the log have locationof the graphical target of the event in the workspace and a time. Also,entries in the log can include a parameter (e.g. url or actual file)identifying graphical constructs used to render the graphical target ona display. Server-side network nodes and client-side network nodes aredescribed which interact to form a collaboration system by which thespatial event map can be made accessible to authorized clients, andclients can utilize the spatial event map to render local display areas,and create events that can be added to the spatial event map and sharedwith other clients.

The workspace associated with a specific collaboration session can berepresented as an unbounded virtual area providing a frame of referencewithout a specified boundary, within which to locate events in time andin virtual collaboration space. The workspace can encompass a virtualarea that is practically unlimited in that it has a size large enoughthat the likelihood of a client-side network node navigating beyond itsboundaries is negligible. For example, a size encompassing a virtualarea that maps to a physical display space including 1,000,000 pixels by1,000,000 pixels can be considered practically unlimited in somesettings. In some examples, the workspace is essentially “infinite” inthat its size is only limited by the extent of the addressing schemeused to identify locations within the virtual space. Also, the systemcan include a number of workspaces, where each workspace can beconfigured individually for access by a single user or by an user group.

The collaboration system can be configured according to an applicationprogram interface API so that the server-side network nodes and theclient-side network nodes can communicate about collaboration events.Messages can be defined that identify events that create or modify agraphical target having a location in the workspace and the time. Theevents can be classified as history events and as ephemeral events,where history events are stored in the spatial event map, and ephemeralevents are not permanently stored with the spatial event map butdistributed among other clients of the collaboration session.

An application program interface (API), including a set of operationsand parameters for the operations, is provided which provides forparticipation in a workspace utilizing a spatial event map. The set ofoperations can be implemented in data processors usingsoftware-implemented functions, which can be hardware-assisted,configured to use the parameters and perform the operations of the API.

The above summary is provided in order to provide a basic understandingof some aspects of the collaboration system described herein. Thissummary is not intended to identify key or critical elements ofinvention or to delineate a scope of invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with respect to specific embodimentsthereof, and reference will be made to the drawings, which are not drawnto scale, and in which:

FIGS. 1A and 1B (collectively FIG. 1) illustrate example aspects of adigital display collaboration environment.

FIG. 2 illustrates a collaboration system including a plurality ofdisplay walls geographically distributed, to which workspace data can bedelivered for use by authorized users.

FIGS. 3 and 4 illustrate aspects of drawing region behavior on thedisplay of FIG. 1.

FIGS. 5A-5E (collectively FIG. 5) is a simplified diagram of datastructures for parts of the workspace data for a workspace.

FIG. 6 illustrates functional features of actors in a workspace in oneexample of a system as described herein.

FIG. 7 is a diagram of a digital display implemented using federateddisplays.

FIG. 8 is a simplified block diagram of the computer system 110, e.g. aclient device computer system (FIG. 1B).

FIG. 9 is a schematic drawing of a database stored accessibly to theclient device computer system 110 (FIG. 1B).

FIG. 10 is a flowchart illustrating aspects of server-side logic of auser login sequence that can be used for a collaboration system.

FIG. 11 is a flowchart illustrating aspects of client-side logic of auser login sequence that can be used for a collaboration system.

FIG. 12 is a flowchart illustrating aspects of client-side logic for adisplay client in a wall being used for a collaboration session.

FIG. 13 is a flowchart illustrating aspects of server-side logicmanaging utilization of distributed display walls in a collaborationsystem.

FIG. 14 is a flowchart illustrating aspects of client-side logic for afederated display system being utilized as a display in a collaborationsystem.

FIG. 15 illustrates, in the style of FIG. 1B, a system supportingdistributed display collaboration where there are displays distributedwidely.

FIG. 16 is a simplified functional block diagram of a client-sidenetwork node and display.

FIG. 17 is a flowchart illustrating operation of a client-side networknode like that of FIG. 16.

FIG. 18 is a flowchart illustrating a procedure executed by logic in aclient-side network node like that of FIG. 16.

DETAILED DESCRIPTION

The following description is presented to enable any person skilled inthe art to make and use the invention, and is provided in the context ofa particular application and its requirements. Various modifications tothe disclosed embodiments will be readily apparent to those skilled inthe art, and the general principles defined herein may be applied toother embodiments and applications without departing from the spirit andscope of the present invention. Thus, the present invention is notintended to be limited to the embodiments shown, but is to be accordedthe widest scope consistent with the principles and features disclosedherein.

The “unlimited workspace” problem includes the need to track how peopleand devices interact with the workspace over time. In order to solvethis core problem, we have created what we call a Spatial Event Map. TheSpatial Event Map contains information needed to define objects andevents in a workspace. It is useful to consider the technology from thepoint of view of space, events, maps of events in the space, and accessto the space by multiple users, including multiple simultaneous users.

Space: In order to support an unlimited amount of spatial informationfor a given collaboration session, we provide a way to organize avirtual space termed the workspace, which can for example becharacterized by a 2-dimensional Cartesian plane with essentiallyunlimited extent in one or both of the dimensions for example, in such away that new content can be added to the space, that content can bearranged and rearranged in the space, that a user can navigate from onepart of the space to another, and that a user can easily find neededthings in the space when it is needed.

Events: Interactions with the workspace are handled as events. People,via tangible user interface devices, and systems can interact with theworkspace. Events have data that can define or point to a targetgraphical object to be displayed on a physical display, and an action ascreation, modification, movement within the workspace and deletion of atarget graphical object, and metadata associated with them. Metadata caninclude information such as originator, date, time, location in theworkspace, event type, security information, and other metadata.

Tracking events in a workspace enables the system to not only presentthe spatial events in a workspace in its current state, but to share itwith multiple users on multiple displays, to share relevant externalinformation that may pertain to the content, and understand how thespatial data evolves over time. Also, the spatial event map can have areasonable size in terms of the amount of data needed, while alsodefining an unbounded workspace.

There can be several different kinds of events in the system. Events canbe classified as persistent events, also referred to as history events,that are stored permanently, or for a length of time required by thesystem for maintaining a workspace during its useful life. Events can beclassified as ephemeral events that are useful or of interest for only ashort time and shared live among other clients involved in the session.Persistent events may include history events stored in an undo/playbackevent stream, which event stream can be the same as or derived from thespatial event map of a session. Ephemeral events may include events notstored in an undo/playback event stream for the system. A spatial eventmap, or maps, can be used by a collaboration system to track the timesand locations in the workspace in some embodiments of both persistentand ephemeral events on workspaces in the system.

Map: A map of events in the workspace can include the sum total ofdiscrete spatial events. When the persistent spatial events for aworkspace are available, then that workspace can be “mapped” to adisplay or screen that has a displayable area of specific size, and thatidentifies a location or area in the workspace to be displayed in thedisplayable area.

Multi-User Access: One key characteristic is that all users, or multipleusers, who are working on a workspace simultaneously, should be able tosee the interactions of the other users in near-real-time way. Thespatial event map allows users having displays at different physicallocations to experience near-real-time events, including both persistentand ephemeral events, within their respective displayable areas, for allusers on any given workspace.

FIG. 1A illustrates example aspects of a digital display collaborationenvironment. In the example, a plurality of users 101 a-h (collectively101), may desire to collaborate with each other in the creation ofcomplex images, music, video, documents, and/or other media, allgenerally designated in FIG. 1A as 103 a-d (collectively 103). The usersin the illustrated example use a variety of devices configured aselectronic network nodes, in order to collaborate with each other, forexample a tablet 102 a, a personal computer (PC) 102 b, and many a largeformat displays 102 c, 102 d, 102 e (collectively devices 102). In theillustrated example the large format display 102 c, which is sometimesreferred to herein as a “wall”, accommodates more than one of the users,(e.g. users 101 c and 101 d, users 101 e and 101 f, and users 101 g and101 h). The user devices, which are referred to as client-side networknodes, have displays on which a displayable area is allocated fordisplaying events in a workspace. The displayable area for a given usermay comprise the entire screen of the display, a subset of the screen, awindow to be displayed on the screen and so on, such that each has alimited area or extent compared to the virtually unlimited extent of theworkspace.

FIG. 1B illustrates the same environment as FIG. 1A. As shown in FIG.1B, the large format displays 102 c, 102 d, 102 e sometimes referred toherein as “walls,” are controlled by respective client-side, physicalnetwork nodes 10, which in turn are in network communication with acentral collaboration server 105 configured as a server-side physicalnetwork node, which has accessible thereto a database 106 storing aspatial event stack for one or more workspaces. As used herein, aphysical network node is an active electronic device that is attached toa network, and is capable of sending, receiving, or forwardinginformation over a communications channel. Examples of electronicdevices which can be deployed as network nodes, include all varieties ofcomputers, work stations, laptop computers, hand held computers andsmart phones. As used herein, the term “database” does not necessarilyimply any unity of structure. For example, two or more separatedatabases, when considered together, still constitute a “database” asthat term is used herein.

The application running at the collaboration server 105 can be hostedusing Web server software such as Apache or nginx. It can be hosted forexample on virtual machines running operating systems such as LINUX. Theserver 105 is heuristically illustrated in FIG. 1B as a single computer.However, the server architecture can involve systems of many computers,each running server applications, as is typical for large-scalecloud-based services. The server architecture includes a communicationmodule which can be configured for various types of communicationchannels, including more than one channel for each client in acollaboration session. For example, near-real-time updates across thenetwork, client software can communicate with the server communicationmodule via using a message-based channel, based for example on the WebSocket protocol. For file uploads as well as receiving initial largevolume workspace data, the client software can communicate with theserver communication module via HTTP. The server can run a front-endprogram written for example in JavaScript served by Ruby-on-Rails,support authentication/authorization based for example on Oauth, andsupport coordination among multiple distributed clients. The servercommunication module can include a message based communication protocolstack, such as a Web Socket application, that performs the functions ofrecording user actions in workspace data, and relaying user actions toother clients as applicable. This system can run on the node.JS platformfor example, or on other server technologies designed to handlehigh-load socket applications.

The database 106 stores, for example, a digital representation ofworkspace data sets for a spatial event map of each session where theworkspace data set can include or identify events related to objectsdisplayable on a display canvas. A workspace data set can be implementedin the form of a spatial event stack, managed so that at leastpersistent spatial events are added to the stack (push) and removed fromthe stack (pop) in a first-in-last-out pattern during an undo operation.There can be workspace data sets for many different workspaces. A dataset for a given workspace can be configured in a database, or as machinereadable document linked to the workspace. The workspace can haveunlimited or virtually unlimited dimensions. The workspace data includesevent data structures identifying objects displayable by a displayclient in the display area on a display wall, and associates a time anda location in the workspace with the objects identified by the eventdata structures. Each device 102 displays only a portion of the overallworkspace. A display wall has a display area for displaying objects, thedisplay area being mapped to a corresponding area in the workspace thatcorresponds to a region in the workspace centered on, or otherwiselocated with, a user location in the workspace. The mapping of thedisplay area to a corresponding area in the workspace is usable by thedisplay client to identify objects in the workspace data within thedisplay area to be rendered on the display, and to identify objects towhich to link user touch inputs at positions in the display area on thedisplay.

The server 105 and database 106 can constitute a server-side networknode, including memory storing a log of events relating to graphicaltargets having locations in a workspace, entries in the log including alocation in the workspace of the graphical target of the event, a timeof the event, and a target identifier of the graphical target of theevent. The server can include logic to establish links to a plurality ofactive client-side network nodes, to receive messages identifying eventsrelating to modification and creation of graphical targets havinglocations in the workspace, to add events to the log in response to saidmessages, and to distribute messages relating to events identified inmessages received from a particular client-side network node to otheractive client-side network nodes.

The logic in the server 105 can comprise an application programinterface, including a specified set of procedures and parameters, bywhich to send messages carrying portions of the log to client-sidenetwork nodes, and to receive messages from client-side network nodescarrying data identifying events relating to graphical targets havinglocations in the workspace.

Also the logic in the server 105 can include an application interfaceincluding a process to distribute events received from one client-sidenetwork node to other client-side network nodes.

The events compliant with the API can include a first class of event(history event) to be stored in the log and distributed to otherclient-side network nodes, and a second class of event (ephemeral event)to be distributed to other client-side network nodes but not stored inthe log.

The server 105 can store workspace data sets for a plurality ofworkspaces, and provide the workspace data to the display clientsparticipating in the session. The workspace data is then used by thecomputer systems 110 with appropriate software 112 including displayclient software, to determine images to display on the display, and toassign objects for interaction to locations on the display surface. Theserver 105 can store and maintain a multitude of workspaces, fordifferent collaboration sessions. Each workspace can be associated witha group of users, and configured for access only by authorized users inthe group.

In some alternatives, the server 105 can keep track of a “viewport” foreach device 102, indicating the portion of the canvas viewable on thatdevice, and can provide to each device 102 data needed to render theviewport.

Application software running on the client device responsible forrendering drawing objects, handling user inputs, and communicating withthe server can be based on HTML5 or other markup based procedures, andrun in a browser environment. This allows for easy support of manydifferent client operating system environments.

The user interface data stored in database 106 includes various types ofobjects including graphical constructs, such as image bitmaps, videoobjects, multi-page documents, scalable vector graphics, and the like.The devices 102 are each in communication with the collaboration server105 via a network 104. The network 104 can include all forms ofnetworking components, such as LANs, WANs, routers, switches, WiFicomponents, cellular components, wired and optical components, and theinternet. In one scenario two or more of the users 101 are located inthe same room, and their devices 102 communicate via WiFi with thecollaboration server 105. In another scenario two or more of the users101 are separated from each other by thousands of miles and theirdevices 102 communicate with the collaboration server 105 via theinternet. The walls 102 c, 102 d, 102 e can be multi-touch devices whichnot only display images, but also can sense user gestures provided bytouching the display surfaces with either a stylus or a part of the bodysuch as one or more fingers. In some embodiments, a wall (e.g. 102 c)can distinguish between a touch by one or more fingers (or an entirehand, for example), and a touch by the stylus. In an embodiment, thewall senses touch by emitting infrared light and detecting lightreceived; light reflected from a user's finger has a characteristicwhich the wall distinguishes from ambient received light. The stylusemits its own infrared light in a manner that the wall can distinguishfrom both ambient light and light reflected from a user's finger. Thewall 102 c may, for example, be an array of Model No. MT553UTBLMultiTaction Cells, manufactured by MultiTouch Ltd, Helsinki, Finland,tiled both vertically and horizontally. In order to provide a variety ofexpressive means, the wall 102 c is operated in such a way that itmaintains “state.” That is, it may react to a given input differentlydepending on (among other things) the sequence of inputs. For example,using a toolbar, a user can select any of a number of available brushstyles and colors. Once selected, the wall is in a state in whichsubsequent strokes by the stylus will draw a line using the selectedbrush style and color.

In an illustrative embodiment, a display array can have a displayablearea totaling on the order of 6 feet in height and 30 feet in width,which is wide enough for multiple users to stand at different parts ofthe wall and manipulate it simultaneously. Flexibility of expression onthe wall may be restricted in a multi-user scenario, however, since thewall does not in this embodiment distinguish between fingers ofdifferent users, or styli operated by different users. Thus if one userplaces the wall into one desired state, then a second user would berestricted to use that same state because the wall does not have a wayto recognize that the second user's input is to be treated differently.

In order to avoid this restriction, the client-side network node candefine “drawing regions” on the wall 102 c. A drawing region, as usedherein, is a region within which at least one aspect of the wall's statecan be changed independently of other regions on the wall. In thepresent embodiment, the aspects of state that can differ among drawingregions include the properties of a line drawn on the wall using astylus. Other aspects of state, such as the response of the system tofinger touch behaviors may not be affected by drawing regions.

FIG. 2 illustrates a distributed collaboration system, which includes ashared server 105 which can be linked to a number of facilities (e.g.facility 1 and facility 2) which are geographically distributed, and atwhich display clients are located. For example Facility 1 may be locatedin New York City, while Facility 2 may be located in Los Angeles. Theremay be many other physical locations at which display clients usable ina collaboration system are located. In this example, Facility 1 includesa first room 151, a second room 152 and a third room 153. Facility 2includes a first room 161, a second room 162, and a third room 163. Thefirst room 151 in Facility 1 includes a large-format display that isimplemented using a plurality of displays. The second room 152 inFacility 1 includes a single screen, intermediate format display. Thethird room 153 in Facility 1 may be a private office or other room inwhich the personal computer or laptop can be utilized as the displayclient for a session interacting in a chosen workspace. Facility 2 inthis illustration is like facility 1, and includes a first room 161, asecond room 162 and a third room 163. The first room 161 in Facility 2includes a large-format display that is implemented using a plurality ofdisplays. The second room 162 in Facility 2 includes a single screen,intermediate format display. The third room 163 in Facility 2 may be aprivate office or other room in which the personal computer, laptop,mobile pad, or mobile phone can be utilized as the display client for asession.

FIG. 2 illustrates a problem that arises in connection with distributedcollaboration systems which rely on large-format or intermediate formatdisplays (or walls) that are located remotely. The large-format andintermediate format displays are not typically under exclusive controlof an individual user. The collaboration server 105 therefore may haveno information about the people having access to the displays at anygiven time.

FIG. 3 illustrates a wall 102 c. The wall in this example is 6 feet talland 30 feet wide. It is initially a default background color or image,and has a default drawing state throughout the wall. Also, workspacedata can define a plurality of objects 301 a to 301 h, collectivelyobjects 301, having locations in the workspace that are mapped tophysical locations in the display area of the wall. The objects 301 cancomprise cards that include text, images or drawing for example, whichare rendered in the display area. Also, the objects 301 can includefeatures allowing a user to interact with the content of the card, orfunctions that are linked to the card using user interface tools at thedisplay, such as the touch screen. Also as illustrated in FIG. 3, adrawing overlay object 302 can be displayed in the display area of thewall.

The drawing state can be a feature of a region independent of objects301, 302 displayed in the region, and is defined by the line drawingproperties, which in the embodiment of FIG. 3 include line appearanceproperties such as brush type, brush size and color. For the purposes ofexample, the system can be configured so that when a user 101 c touchesthe wall, using either a stylus or one or more fingers (sometimesreferred to collectively herein as a writing implement), a toolbar 210appears nearby and a drawing region 212 is defined. Touching a touchpoint is one embodiment of what is sometimes referred to herein as“opening user input”; other embodiments will be apparent to the reader.The initial drawing state of a newly defined drawing region is apredefined default (such as brush type=ink, thickness=5 mm,color=white), which in various embodiments may or may not match thedefault state of the remainder of the wall. In the embodiment of FIG. 2the drawing properties established for a drawing region apply throughoutthe drawing region. Line drawing operates on the wall logically in alayer above any application program that might be running on thecomputer system 110, regardless of whether the program has ownership ofany particular area of the wall 102 c.

In the embodiment of FIG. 3 drawing regions always fill the entirevertical extent of the wall, though in other embodiments regions can beshorter, and/or have non-rectangular shapes. Also in the embodiment ofFIG. 3 drawing regions are perceptibly demarcated with left and righthand borders 214 and 216; in another embodiment other means may be usedto demarcate the region, such as background shading. In yet anotherembodiment the region boundaries are not perceptible to the user.Assuming sufficient space to the left and right, the client-sidecomputer system 110 can spawn the drawing region in a position that iscentered about the user's touch point. Drawing regions have a minimumwidth Wmin and an ideal width Wideal. The minimum width preferably ischosen to be the smallest width to allow reasonably unfetteredexpression, and in the embodiment of FIG. 3 is 4 feet. The ideal widthpreferably is chosen to be roughly equal to the widest span of anaverage user's arms stretched out horizontally, and in the embodiment ofFIG. 3 is 6 feet.

If there is plenty of space on either side of the user's touch point,then the computer system 110 can set the initial region width to Wideal.This is the scenario illustrated in FIG. 3. If the user's touch point istoo close to a wall edge for a new drawing region to be centered aboutit, then the computer system 110 will abut the new drawing regionagainst the wall edge. The new drawing region will still have a widthWideal assuming sufficient space is available, so the new drawing regionwill not be centered about the user's touch point. On the other hand, ifthe user's touch point is far enough from the wall edge to create adrawing region centered about the touch point, but the new drawingregion would be less than Wmin from the wall edge, then the gap spacebetween the wall edge and the new drawing region is considered unusable.In this case the computer system 110 will extend the new drawing regionto fill up the unusable space.

FIG. 4 illustrates a scenario, for the purposes of example, in which twousers 101 c and 101 d can use the wall simultaneously. Initially, user101 c touches the wall 102 c at touch point 516, and in response theretothe computer system 110 spawns drawing region 512 with toolbar 510.Optionally, user 101 c then touches controls on toolbar 510 in order tochange the line appearance properties within region 512. Next, a seconduser 101 d touches the wall 102 c at touch point 518, which is withinthe wall 102 c background (i.e. outside of all pre-existing drawingregions). A second drawing region 514 is then spawned by the computersystem 110, with toolbar 520. If user 101 d draws a line at this timewithin region 514, the computer system 110 will paint it with thedefault line properties rather than those previously set by user 101 cfor drawing region 512. User 101 d then optionally touches controls ontoolbar 520 in order to change the line appearance properties withinregion 514. Subsequent lines drawn in region 514 will then adopt the newline appearance properties. The line appearance properties of region 512will remain unchanged.

Drawing regions can also be made to automatically track the movement ofthe stylus. Although numerous possible tracking algorithms will beapparent to the reader, one that follows these minimum rules ispreferred: (1) the region does not move so long as the stylus remainsrelatively near the center of the region; and (2) as the stylusapproaches a region boundary, the region moves so that the boundaryremains ahead of the stylus.

Drawing regions provide one example of user interaction that can have aneffect at a local display wall, but not have an effect on the globalworkspace data. As illustrated in this example, the locations of theobjects 301, 302 are not affected by the assignment of drawing regions,the toolbars, and the drawing overlays within the regions. Of course inother types of user interface interactions, the locations of the objects301, 302 can be moved, and such movements can be events related toobjects in the global workspace data.

A variety of behaviors related to the interpretation of user input basedon interaction with a local wall are described in co-pending U.S.application Ser. No. 13/758,984, filed on 4 Feb. 2013, entitled REGIONDYNAMICS FOR DIGITAL WHITEBOARD, now U.S. Pat. No. 9,479,548, which isincorporated by reference above. These behaviors are illustrative oflocal processing of user input and image data at a wall that can beexecuted by the local computer systems 110, with little or no effect onthe shared workspace data maintained at the collaboration server in someembodiments.

FIGS. 5A-5E represent data structures which can be part of workspacedata maintained by a database at the collaboration server 105. In FIG.5A, an event data structure is illustrated. An event is an interactionwith the workspace data that can result in a change in workspace data.Thus an event can include an event identifier, a timestamp, a sessionidentifier, an event type parameter, the client identifier, and an arrayof locations in the workspace, which can include one or more for thecorresponding event. It is desirable for example that the timestamp haveresolution on the order of milliseconds or even finer resolution, inorder to minimize the possibility of race conditions for competingevents affecting a single object. Also, the event data structure caninclude a UI target, which identifies an object in the workspace data towhich a stroke on a touchscreen at a client display is linked. Eventscan include style events, which indicate the display parameters of astroke for example. The events can include a text type event, whichindicates entry, modification or movement in the workspace of a textobject. The events can include a card type event, which indicates thecreation, modification or movement in the workspace of a card typeobject. The events can include a stroke type event which identifies alocation array for the stroke, and display parameters for the stroke,such as colors and line widths for example.

Events can be classified as persistent, history events and as ephemeralevents. Processing of the events for addition to workspace data, andsharing among users can be dependent on the classification of the event.This classification can be inherent in the event type parameter, or anadditional flag or field can be used in the event data structure toindicate the classification.

A spatial event map can include a log of events having entries forhistory events, where each entry comprises a structure such asillustrated in FIG. 5A. The server-side network node includes logic toreceive messages carrying ephemeral and history events from client-sidenetwork nodes, and to send the ephemeral events to other client-sidenetwork nodes without adding corresponding entries in the log, and tosend history events to the other client-side network nodes while addingcorresponding entries to the log.

FIG. 5B illustrates a card data structure. The card data structure canprovide a cache of attributes that identify current state informationfor an object in the workspace data, including a session identifier, acard type identifier, an array identifier, the client identifier,dimensions of the cards, type of file associated with the card, and asession location within the workspace.

FIG. 5C illustrates a data structure which consolidates a number ofevents and objects into a catchable set called a chunk. The datastructure includes a session ID, and identifier of the events includedin the chunk, and a timestamp at which the chunk was created.

FIG. 5D illustrates the data structure for links to a user participatingin a session in a chosen workspace. This data structure can include anaccess token, the client identifier for the session display client, theuser identifier linked to the display client, a parameter indicating thelast time that a user accessed a session, and expiration time and acookie for carrying various information about the session. Thisinformation can for example maintain a current location within theworkspace for a user, which can be used each time that a user logs in todetermine the workspace data to display at a display client to which thelogin is associated.

FIG. 5E illustrates a display array data structure which can be used inassociation with large-format displays that are implemented by federateddisplays, each having a display client. The display clients in suchfederated displays cooperate to act as a single display. The workspacedata can maintain the display array data structure which identifies thearray of displays by an array ID, and identifies the session position ofeach display. Each session position can include an x-offset and ay-offset within the area of the federated displays, a sessionidentifier, and a depth.

The system can encrypt communications with client side network nodes,and can encrypt the database in which the spatial event maps are stored.Also, on the client-side network nodes, cached copies of the spatialevent map are encrypted in some embodiments, to prevent unauthorizedaccess to the data by intruders who gain access to the client-sidecomputers.

FIG. 6 is a diagram representing a functional architecture for adistributed collaboration system used to create, modify, distribute anddisplay workspace data for a workspace. The basic configuration includesa collaboration service 601 which manages event data executed by aserver, such as server 105, a portal service 602 which can be executedby a server such as server 105 or located in other computer systemsaccessible to the server, such as a peer network node, and a displayclient 603 located at a client-side network node, at which the userinteraction is active. The display client 603 is in communication withthe collaboration service 601 and with the portal 602. The communicationchannel 613 between the display client 603 and a collaboration service601 manages the download of session history, and the live update ofsession events. Also, across this channel 613, a display client 603 canupload images that can be associated with events to the collaborationservice 601. The display client 603 is in communication with the portal602 across communication channel 623. The portal 602 to manages ahomepage for the workspace data, session management and useradministration. This portal can be utilized for user login,authentications, and for delivering image files and the like as analternative to, and in parallel with, the communication channel 613. Thecollaboration service 601 and portal 602 are in communication acrosschannel 612. The collaboration service 601 and portal 602 manageauthentication and authorization protocols, and coordinate sessionadministration, and workspace data management.

The display client 603 can be part of a client-side network nodeincluding a physical or virtual computer system having computer programsstored in accessible memory that provide logic supporting thecollaboration session, including an HTML 5 client, wall arraycoordination logic for display array implementations, workspace dataparsing searching and rendering logic, and a session events applicationto manage live interaction with workspace data at the server and thedisplay wall.

The portal 602 can be part of a server-side network node including aphysical or virtual computer system having computer programs stored inaccessible memory, that provide logic supporting user access to thecollaboration server. The logic can include applications to provideinitial entry points for users, such as a webpage with login resources,logic to manage user accounts and session anticipation, logic thatprovides authorization services, such as OAuth-based services, andaccount data.

The collaboration service 601 can be part of a server-side network nodeincluding, and can manage the session event data, coordinate updatedevents among clients, deliver catchable history and images to clients,and control access to a database stored in the workspace data.

FIG. 7 illustrates an optional technology for implementation of adisplay wall, based on a display implemented by a plurality of displays701-704 with federated control. In this example, each display 701-704 isassociated with a corresponding display client 711-714. Each displayclient can execute a browser used to render objects from the workspaceon the display area, which has a plurality of subsets of display areawhich correspond to each the plurality of displays. Each display clientcan be configured to manage display in a subset of the display area forthe session, for example by storing an offset parameter (e.g. 0, 0 fordisplay 701; 0, 1 for display 702; 1, 0 for display 703; and 1, 1 fordisplay 704).

Each of the display clients 711-714 can maintain a communication channel721-724 with the collaboration server 105, which is in turn coupled tothe workspace database 106. The collaboration server 105 and/or theclient can maintain a user location within the workspace for eachauthorized user. When an authorized user is logged in, and has selecteda display array such as that shown in FIG. 7 as the display canvas, thecollaboration server can link each of the display clients 711-714 into agroup associated with the session and user. The collaboration server canthen download a current user location within the workspace to thedisplayable area, or canvas, for each of the display clients in thegroup. The display clients in the group can independently apply theiroffset parameter to identify session locations to map onto the subset ofthe workspace indicated by the offset parameter. In an alternative, thecollaboration server can manage the offset computation in communicationwith each of the display clients 711-714 by delivering to each clientthe current user location as offset according to the arraycharacteristics.

In order to support coordination of a single display among a pluralityof displays, each of the display clients 711-714 can also communicatewith each of the other display clients with events that are local to themanagement of the display area, and which do not have an effect on theglobal workspace data. Alternatively, the display client 711-714 cancommunicate solely with the collaboration server 105, which can thendirect local events back to the group of display clients associated withthe session, and global events to all of the display clients in activesessions with the workspace, and to the database storing workspace data.

The display clients at a single display comprised of federated displayscan be implemented individual computer systems coupled to thecorresponding displays, or can be implemented using a single computersystem with virtual machines coupled to the corresponding displays.

Also, a single display driver can be configured to control the entiresurface of a collection of physical displays arranged as a singledisplay wall.

A spatial event map system can include an API executed in coordinationby client-side and server-side resources including any number ofphysical and virtual machines. One example of an API is described below.An API can be defined in a variety of ways, while including the elementssupporting maintenance of a spatial event map in a server-side networknode or nodes, and supporting sharing of the spatial event map with oneor a plurality of active client-side network nodes. In this example, theAPI is broken down in this example into processes managed by twoservers:

Socket Requests Server (Websockets)—used for updating clients withrelevant data (new strokes, cards, clients, etc.) once connected. Alsohandles the initial connection handshake.

Service Requests Server (HTTP/REST)—used for cacheable responses, aswell as posting data (i.e. images and cards)

Client-side network nodes are configured according to the API, andinclude corresponding socket requests clients and service requestsclients.

Socket Requests:

The socket server can execute a network protocol that maintainsconnections via Websockets. The messages used in the API can beencapsulated within the Websocket protocol. Messages can be individualUTF-8 encoded JSON arrays.

Initial loading of history, including all or part of a spatial event mapof a collaboration session, at the client-side network nodes can be doneusing HTTP requests via the Service Requests Server, rather thanwebsockets to support caching.

Socket Connection

http://localhost:4545/<sessionId>/socket?device=<device>

sessionId - - (string) the id of the session to join

device - - (string) a device type, such as a wall or a desktop.

Message Structure

The first element of each message array is a sender-id, specifying theclient that originated the message. Sender-ids are unique among allsessions on the server. The id and cr messages sent from the server tothe client have their sender-id set to a default value, such as −1. Thesecond element of each message array is a two-character code. This codedefines the remaining arguments in the array as well as the intendedaction. Messages sent with a sender-id of −1 are messages that originatefrom the server.

Valid Message Types

The following are messages supported by the API example herein. Many ofthese messages take the following parameter:

sender-id: - - the ID of the client sending the message, or −1 if themessage originates with the server.

Client ID Request:

// server <--client [“id”, sessionId, zoomLevel, x1, y1, x2, y2]

This request can be used to enable interaction with the socket API.

This request starts the asynchronous client-id request/responsehandshake. The next section explains the asynchronous acknowledgment ofthe new client (including the provided client ID).

SessionId - - (string) the id of the workspace to join.

zoomLevel - - (int) the zoom level desired by this client

x1, y1 - - (int, optional) the desired point of origin for the usersviewport

x2, y2 - - (int, optional) the desired point of extent for the usersviewport

There is no sender-id sent with this message.

The zoom level sent in this message is the zoom level preferred by thisclient. If the client joins an empty display array (via the “id”message), the client's preferred zoom level becomes the initial zoomlevel for the display array. If the client joins an existing displayarray, the preferred zoom level sent in its “id” message is ignored, andthe zoom level associated with the existing display array is sent (inthe “av” message).

Client ID Response:

// server -->client [−1, “id”, client-id]

Clients are required to store the assigned client ID for use insubsequent socket requests.

Informs a new client of their ID. In this case, sender-id is set to −1

client-id - - (string) the ID of the newly-joined client

Join Room:

// server <-- client [sender-id, “jr”, room-id, [data]] [sender-id,“jr”, “lobby”][sender]Informs the server of an attempt by the client to join a room.

room-id - - can contain one of lobby or session.

data - - is a wildcard set of arguments, which should be used toinitialize the room.

Room Data Arguments:

Session requires “session-id” containing the id of the session. Arrayrequires:

-   -   arrayId - - (string) id of the display array    -   x - - (integer) x offset of this display    -   y - - (integer) y offset of this display    -   width - - (integer) width of this display    -   height - - (integer) height of this display        Server will respond with a “room” message in response.

Room Join Response:

// server -->client [−1, “room”, [room-id], [databag]] [−1, “room”,“lobby”, {pin: pin}]

room-id - - contains one of: lobby or session

databag - - is a room-specific bag of variables:

lobby provides:

-   -   pin - - containing the pin for wall authentication

session provides:

-   -   sessionName - - containing the name of the session

Room List Response

// server -->client [−1, “rl”, roomMembershipList]

Informs the client of the room memberships. Room memberships includeinformation regarding clients visiting the same room as you.

roomMembershipList - - (array of room membership objects)

Session Request:

// server <-- client [sender-id, “sr”]

Informs the server that the client would like the list of joinableactive sessions.

Session List:

// server -->client [−1, “sl”, sessionList]

Informs the client of the joinable active sessions.

SessionList - - (array of strings)

Object ID Reservation:

Use this to create a new unique object id that is acceptable forcreating new history events which create an object.

// server <->client [sender-id, “oid”]

Server responds with:

[−1, ‘oid’, <new-object-id>]

History Event

All persistent events are sent as HistoryEvent. This includes: ** movingwindows ** setting text ** deleting windows ** creating windows.

HistoryEvents are written to the session's history and returned when thehistory is retrieved.

HistoryEvents are sent to the server without an eventId. The serverassigns an eventId and broadcasts the event to all clients (includingthe originating client).

New object ids can be reserved using the oid message.

Basic Message Format

// server <-- client [client-id, “he”, target-id, event-type,event-properties]

client-id - - (string) the ID of the originating client

target-id - - (string) the ID of the target object/widget/app to whichthis event is relevant

event-type - - (string) an arbitrary event type

properties - - (object) a JSON object describing pertinent key/valuesfor the event.

// server -->client[client-id, “he”, target-id, event-id, event-type,event-properties

client-id - - (string) the ID of the originating client

target-id - - (string) the ID of the target window to which this eventis relevant

event-id - - (string) the ID of the event in the database

event-type - - (string) an arbitrary event type

properties - - (object) a JSON object describing pertinent key/valuesfor the event.

Example Interaction: Moving Objects

A good example illustrating some of the persistent event/ephemeral eventclassification is moving an object. While the object is being moved orfor example resized by dragging, a series of ephemeral events (termed“volatile events VEs”) is sent to the server, and re-broadcast to allclients in the session. Once the user finishes moving the object, theclient should send a history event to specify the rect and order of theobject:

[“511d6d429b4aee0000000003”,“he”,“511d619c9b4aee0000000039”,“position”,{“rect”}. . .

The server will respond with the newly persisted HE record. Note theinclusion of the record's eventId.

// server->client format of ‘he’ is: [<clientId>, <messageType>,<targetId>, <eventId>,

Note: The eventId will also be included in history that is fetched viathe HTTP API.

History Events by Object/Application Type

Session

Create - - Add a note or image on the work session

stroke - - Add a pen or eraser stroke on the background

Note

text - - Sets or update the text and/or text formatting of a note.

delete - - Remove the note from the work session

position - - Update the size or location of the note in the work session

pin - - Pin or unpin the note

stroke - - Add a pen or eraser stroke on top of the image

Image

delete - - Remove the note from the work session

position - - Update the size or location of the note in the work session

pin - - Pin or unpin the note

stroke - - Add a pen or eraser stroke on top of the image

History Event Details

Text

sets and styles the text of a note. Both the text attribute and styleattribute are optional.

// server <-- client[client-id, “he”, target-id, “text”, {“text”:“abcdef”,

Create

sent to clients when the server receives a card create (cc) message oran image upload.

For create messages the target-id is the session-id.

// server -->client[client-id, “he”, session-id, event-id, “create”,

{“id”:“5123e7ebcd18d3ef5e000001”

Properties

id - - (int) unique identifier for the window

baseName - - (string) the background image file name

ext - - (string) the background image file extension

rect - - (object) the location of the window

actualWidth - - (int) the background image width

actualHeight - - (int) the background image height

order - - (int) z order

type - - (string) “note” for objects that can have text, “image” forother objects

regionId - - (string) the canvas region if the object is created in acanvas region

hidden - - (boolean) whether the window is currently hidden

Delete

used to make a window disappear from the session. Delete is an undo-ableaction.

// server <-- client[client-id, “he”, target-id, “delete”,{“hidden”:true}]// server -->

Position

used to save the position of a window after a move, fling, or resize

// server <-- client[client-id, “he”, target-id, “position”,{“rect”:[−1298,−390,−1018

Properties

-   -   rect - - (object) the location of the target window    -   order - - (int) the z-order of the target window

Stroke

used to save a stroke

/ server <-- client[client-id, “he”, target-id, “stroke”, {“size”: 10,“brush”:

Properties

-   -   locs - - (array) stroke locations in the format: [10, 1, 10, 2,        12, 3] where coordinates are paired [x, y, x, y, x, y] in an        array

Pin

sent to clients to pin a note or image in place or to remove an existingpin. Windows that are pinned cannot be moved or resized until they areunpinned.

// server -->client[client-id, “he”, session-id, event-id, “pin”,{“pin”: true}]

Properties

-   -   pin - - (boolean) true is pin, false is un-pin        Volatile Event

Volatile events are ephemeral events not recorded in the undo/playbackevent stream, so they're good for in-progress streaming events likedragging a card around the screen, and once the user lifts their finger,a HistoryEvent is used to record its final place.

// server <-->client[client-id, “ve”, target-id, event-type,event-properties]

client-id - - (string) the ID of the originating client

target-id - - (string) the ID of the target window to which this eventis relevant

event-type - - (string) an arbitrary event type

properties - - (object) a JSON object describing pertinent key/valuesfor the event.

Volatile Events by Object/Application Type

Session

sb - - Starts a stroke. Used to render strokes on one client while theyare being drawn on another client.

sc- - Continues a previously started stroke by giving another point toinclude. Used to render strokes while they are being drawn on anotherclient.

se - - Ends a previously started stroke.

Note

fling- - Animates a note sliding from one place in the work session toanother. This is the visual response to a flick or fling action by theuser.

position- - Live updates the position of a note while its being moved byanother user.

sb- - Starts a stroke. Used to render strokes on one client while theyare being drawn on another client.

sc - - Continues a previously started stroke by giving another point toinclude. Used to render strokes while they are being drawn on anotherclient.

se - - Ends a previously started stroke.

Image

fling - - Animates an image sliding from one place in the work sessionto another. This is the visual response to a flick or fling action bythe user.

position - - Live updates the position of an image while its being movedby another user.

sb - - Starts a stroke. Used to render strokes on one client while theyare being drawn on another client.

sc - - Continues a previously started stroke by giving another point toinclude. Used to render strokes while they are being drawn on anotherclient.

se - - Ends a previously started stroke.

Types of Volatile Events

Fling

used to broadcast a fling action to all connected clients.

// server <-->client[client-id, “ve”, target-id, “fling”, {“velocityX”:10, “velocityY”

Properties

-   -   velocityX (int) the x component of the fling vector    -   velocityY (int) the y component of the fling vector

position—ve

used to broadcast intermediate steps of a window move

// server <-->client[client-id, “ve”, target-id, “position”,{“rect”:[−1298,−390,−1018

Properties

-   -   rect (object) the location of the target window    -   order (int) the z-order of the target window

sb:

used to broadcast the beginning of a stroke

// server <-->client[client-id, “ve”, target-id, “sb”, {“brush”:1,“size”:2, “color”

Properties

-   -   x,y - - (int) the starting point of this stroke    -   strokeId - - (string) the ID of the new stroke

sc:

// server <-->client[client-id, “ve”, target-id, “sc”, {“x”:100,“y”:300, “strokeId”

used to broadcast a continuation of a stroke

Properties

-   -   x,y - - (int) the new end-point of the stroke    -   strokeId - - (string) the ID of the new stroke

se:

// server <-->client[client-id, “ve”, target-id, “se”, “strokeId”:“395523d316e942b496a2c8a6fe5f2cac”

End the stroke specified by stroke-id

stroke-id - - (string) the ID of the continued stroke

Delete Stroke:

// server -->client[sender-id, “sd”, stroke-id, target-id]

Delete a stroke.

stroke-id - - (string) the ID of stroke

target-id - - (string) the ID of the stroke target

Undo:

Undoes the last undo-able action (move, set text, stroke, etc).

// server <-- client

[sender-id, “un”]// server -->client

[client-id, ‘undo’, target-

Undo Example: Move a window and then undo that move

The following example shows a move, and how that move is undone.

// Client sends move and then an undo message.

The server removes the history event of the move from the sessionhistory and notifies the client that this record will no longer be apart of the session's spatial event map, taking it out of the historytimeline. Future requests of the history via the HTTP API will notinclude the undone event (until after a redo).

Display Array Dimensions:

// server -->client [−1, “dd”, arrayId, width, height]

Informs clients of changes to the overall display array width andheight. This may not be utilized with the client-side network node hasresources to manage the local display of portions of the spatial eventmap.

arrayID - - (string) the ID of the display array

width, height (integers) width and height of the display array in pixels

Pan Array:

// client -->server [sender-id, “pa”, newArrayOffsetX, newArrayOffsetY]Inform the server of a pan to a new location.

newArrayOffsetX, newArrayOffsetY (int) the new location of the displayarray after panned.

Session Change:

// server -->client [sender-id, “cs”, sessionId]

Inform siblings in a display array that the session has changed.

SessionId - - (string) is the id of the session to switch to

Zoom Change:

// client -->server [sender-id, “zc”, zoomLevel, zoomCenterX,zoomCenterY]

Inform the server that a zoom was requested.

zoomLevel (integer) the zoom level to transition to, from 1 to 11

zoomCenterX (integer) the x coordinate of the origin of the zoom

zoomCenterY (integer) the y coordinate of the origin of the zoom

Map-Mode Change:

// client -->server

[sender-id, “mm”, zoomLevel, zoomCenterX, zoomCenterY]

Inform the server that map-mode was requested. Superficially, thisoperates near identically to the zoomchange message, except where dozensor hundreds of zoom-change messages are meant to be sent rapid-fire withtweening between them in the graphical treatment, the map-mode messageis intended for a single zoom snap with different transition effects.

zoomLevel - - (integer) the zoom level to transition to, from 1 to 11

zoomCenterX - - (integer) the x coordinate of the origin of the zoom

zoomCenterY - - (integer) the y coordinate of the origin of the zoom

Create Card

// server <-- client

[sender-id, “cc”, templateId, regionId, x, y, x2, y2]

templateId - - (string) the id of the card template to be used

regionId (string) the canvas region id of the originating event (if any)

x, y, x2, y2 - - (int) the desired rect for the new card

User Permissions

// server -->client [sender-id, “up”, permissions]

permissions a hash of permission types and true/false to indicate if theauthenticated user has that permission. Currently the only permission is“can_share” indicating users who can share the session with others.

Save Position:

// client -->server [−1, “sp”, zoomLevel, x, y]

Saves the current screen position. On reconnect, the client will receivea ‘zc’ (zoom-change) and ‘pa’ (pan-array) message sending them back tothis location.

zoomLevel (integer) the zoom level the device is currently on

x (integer) the x coordinate of the origin of the screen

y (integer) the y coordinate of the origin of the screen

Stroke IDs

Stroke ID's are selected by the client. Currently they are the sender-idcomposed with an increasing integer, separated by a dot. This is to makeit unique within the server context among all clients.

Target IDs

A stroke may be attached to a specific target in the session, like asub-canvas (called a “card”) that floats above the main canvas. In thecase of a stroke belonging to a card, the target ID field would containthe card ID. Strokes destined for the main canvas in the session aredesignated by having their target ID be the same as the session name.

Establishing Connections

When a client establishes a new websocket connection with the server,the server first chooses a unique client ID and sends it in an “id”message to the client. It then sends the “pa” message with senderid setto −1.

The representative flow then is for the client to perform an HTTP GET“/:sessionId/objects” (documented below) to receive information aboutthe cards in the session. Then the client requests “/:sessionId/history”(also documented below), which receives an array of history urls. Theseare broken into batches to improve cachablity.

The client then GETs each url not stored in the local cache, and theserver will respond with the stroke data for those history batches.

Service Requests

History:

Gets a list of history bookmarks. Each bookmark is a span of cachedstroke history.

curl http://localhost:4545/<sessionId>/history

sessionId

name of the session you're getting the history for

Response Headers

HTTP/1.1 200 OKX-Powered-By: ExpressAccess-Control-Allow-Origin:*Access-Control-Allow-Headers:

Response

[“/<sessionId>/history/<startTime>/<endTime>?b=1”][“/<sessionId>/history/<startTime>/<endTime>?

sessionId - - (string) id of the session to switch to

startTime - - (integer) beginning timestamp

endTime - - (integer) ending timestamp

b - - cache buster

Retrieving a Block of History:

Gets the history between start time and end time. A request needs to bemade for each returned span of history.

curlhttp://localhost:4545/<sessionId>/history/<startTime>/<endTime>?b=<cache-buster>

sessionId - - id of the session you're getting the history for

startTime - - the start time as given by initial history request

endTime - - the end time as given my initial history request

cacheBuster - - a simple key that will be changed whenever client-storedcache is no longer valid

Response Header

-   -   HTTP/1.1 200 OKX-Powered-By: ExpressAccess-Control-Allow-Origin:        *Access-Control-Allow-Headers: X-Requested-With Content-Type:        application/json    -   Content-Length: 2134    -   ETag: 1346968307576    -   Date: Fri, 14 Sep. 2012 17:35:14 GMT    -   Connection: keep-alive        Response

[ [ 4, “sx”, “4.4”, [537, 650, 536, 649, 536, 648, ...], { “size”: 10,“color”: [0, 0, 0, 1], “brush”: 1 }, 1347644106241, “cardFling” ] ] (seedocumentation for sx “stroke-complete” websocket message)

Retrieving Objects:

Gets the objects (cards/images) for the requested session.

curl http://localhost:4545/<sessionId>/objects

sessionId id of the session you're getting the history for

Response Header

HTTP/1.1 200 OK

X-Powered-By: Express

Access-Control-Allow-Origin: *

Access-Control-Allow-Headers: X-Requested-With

Content-Type: application/json; charset=utf-8

Content-Length: 285

Date: Fri, 14 Sep. 2012 17:35:14 GMT

Connection: keep-alive

Response

[ { “eventType”: “oc”, “id”: “50536840ce64b39439000005”, “baseName”:“sessions/all/green”, “ext”: “JPEG”, “rect”: [−239, 49, 361, 649],“arrayId”: 3, “clientId”: 3, “regionId”: null, “sessionId”: “cardFling”,“actualWidth”: 600, “actualHeight”: 600, “order”: null, “_id”:“50536840ce64b39439000005”, “type”: “image” }, { “eventType”: “oc”,“id”: “50536b66ce64b39439000006”, “baseName”: “sessions/all/orange”,“ext”: “JPEG”, “rect”: [−97, 190, 503, 790], “arrayId”: 5, “clientId”:5, “regionId”: null, “sessionId”: “cardFling”, “actualWidth”: 600,“actualHeight”: 600, “order”: null, “_id”: “50536b66ce64b39439000006”,“type”: “image” } ]

Card Templates:

Gets a list of global card templates for creating cached, re-usablecards. This is different from uploading a file as the samebackground-image is used for all cards created with this template.

curl http://localhost:4545/card templates.json

Response

[ { “id”:“50901cb0b9a18c190902a938”, “width”:600,“thumbnail”:“card_templates/thumbnails/pink.jpeg” }, {“id”:“50901cb0b9a18c190902a939”, “width”:600,“thumbnail”:“card_templates/thumbnails/green.jpeg” } ]

These values can be used to send a create card message:

// creates a new card using the pink template above

[“cc”, “50901cb0b9a18c190902a938”, <regionIdOrNull>, <x>, <y>]

Upload:

Sends an image to the server to be placed in the session.

-   -   curl-F “file=@photo.JPG”-F “x=236”-F “y=832”-F “clientId=10”-F        “sessionId=cardFling”    -   -F “arrayId=10”-F “order=23”-F “x2=899”-F “y2=1495”-F        “filename=photo.jpg”    -   http://localhost:4545/<sessionId>/object/upload

Params

x: x position of drop

y: y position of drop

clientId: client Id

sessionId: session Id

arrayId: array identifier

order: z order

x2: x position of bottom right corner of drop

y2: y position of bottom right corner of drop

filename: name of file uploaded

The API describe above provides one example message structure. Otherstructures may be utilized as well, as suits a particularimplementation.

In an embodiment of the collaboration system, an application programinterface API is executed by the collaboration server 105 and displayclients based on two communication channels for each display client, assuggested with reference to FIG. 6.

In some embodiments a federated display may be deployed. In a federateddisplay, each of the display clients 711-714 can independently maintaintwo channels with the server. The first channel is a message basedsystem configured for communications about live events. In one example,this first channel is implemented using a set of socket requests over aWebsocket channel with the collaboration service 601, and used by theserver for updating clients with relevant data (new strokes, cards,clients, etc.) once connected. The message based channel can also handlethe initial connection handshake. A second channel is a more statelesstype link with the portal 602, such as n HTTP/REST interface, which canbe used for cacheable responses, as well as posting data (i.e. imagesand cards). Also an initial loading of workspace data to a displayclient can be done using HTTP requests rather than the message basedchannel (Websockets) to support caching.

FIG. 8 is a simplified block diagram of a computer system, or networknode, which can be used to implement the client-side functions (e.g.computer system 110) or the server-side functions (e.g. server 105) in adistributed collaboration system. A computer system typically includes aprocessor subsystem 1014 which communicates with a number of peripheraldevices via bus subsystem 1012. These peripheral devices may include astorage subsystem 1024, comprising a memory subsystem 1026 and a filestorage subsystem 1028, user interface input devices 1022, userinterface output devices 1020, and a network interface subsystem 1016.The input and output devices allow user interaction with the computersystem. Communication module 1016 provides physical and communicationprotocol support for interfaces to outside networks, including aninterface to communication network 104, and is coupled via communicationnetwork 104 to corresponding communication modules in other computersystems. Communication network 104 may comprise many interconnectedcomputer systems and communication links. These communication links maybe wireline links, optical links, wireless links, or any othermechanisms for communication of information, but typically it is anIP-based communication network, at least at its extremities. While inone embodiment, communication network 104 is the Internet, in otherembodiments, communication network 104 may be any suitable computernetwork.

The physical hardware component of network interfaces are sometimesreferred to as network interface cards (NICs), although they need not bein the form of cards: for instance they could be in the form ofintegrated circuits (ICs) and connectors fitted directly onto amotherboard, or in the form of macrocells fabricated on a singleintegrated circuit chip with other components of the computer system.

User interface input devices 1022 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touch screen incorporated into the display (including thetouch sensitive portions of large format digital display 102 c), audioinput devices such as voice recognition systems, microphones, and othertypes of tangible input devices. In general, use of the term “inputdevice” is intended to include all possible types of devices and ways toinput information into the computer system or onto computer network 104.

User interface output devices 1020 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. In theembodiment of FIG. 1B, it includes the display functions of large formatdigital display 102 c. The display subsystem may also provide non-visualdisplay such as via audio output devices. In general, use of the term“output device” is intended to include all possible types of devices andways to output information from the computer system to the user or toanother machine or computer system.

Storage subsystem 1024 stores the basic programming and data constructsthat provide the functionality of certain embodiments of the presentinvention.

The storage subsystem 1024 when used for implementation of server sidenetwork-nodes, comprises a product including a non-transitory computerreadable medium storing a machine readable data structure including aspatial event map which locates events in a workspace, wherein thespatial event map includes a log of events, entries in the log having alocation of a graphical target of the event in the workspace and a time.Also, the storage subsystem 1024 comprises a product includingexecutable instructions for performing the procedures described hereinassociated with the server-side network node.

The storage subsystem 1024 when used for implementation of client sidenetwork-nodes, comprises a product including a non-transitory computerreadable medium storing a machine readable data structure including aspatial event map in the form of a cached copy as explained below, whichlocates events in a workspace, wherein the spatial event map includes alog of events, entries in the log having a location of a graphicaltarget of the event in the workspace and a time. Also, the storagesubsystem 1024 comprises a product including executable instructions forperforming the procedures described herein associated with theclient-side network node.

For example, the various modules implementing the functionality ofcertain embodiments of the invention may be stored in storage subsystem1024. These software modules are generally executed by processorsubsystem 1014.

Memory subsystem 1026 typically includes a number of memories includinga main random access memory (RAM) 1030 for storage of instructions anddata during program execution and a read only memory (ROM) 1032 in whichfixed instructions are stored. File storage subsystem 1028 providespersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD ROM drive, an optical drive, or removable media cartridges. Thedatabases and modules implementing the functionality of certainembodiments of the invention may have been provided on a computerreadable medium such as one or more CD-ROMs, and may be stored by filestorage subsystem 1028. The host memory 1026 contains, among otherthings, computer instructions which, when executed by the processorsubsystem 1014, cause the computer system to operate or performfunctions as described herein. As used herein, processes and softwarethat are said to run in or on “the host” or “the computer,” execute onthe processor subsystem 1014 in response to computer instructions anddata in the host memory subsystem 1026 including any other local orremote storage for such instructions and data.

Bus subsystem 1012 provides a mechanism for letting the variouscomponents and subsystems of a computer system communicate with eachother as intended. Although bus subsystem 1012 is shown schematically asa single bus, alternative embodiments of the bus subsystem may usemultiple busses.

The computer system itself can be of varying types including a personalcomputer, a portable computer, a workstation, a computer terminal, anetwork computer, a television, a mainframe, a server farm, or any otherdata processing system or user device. In one embodiment, a computersystem includes several computer systems, each controlling one of thetiles that make up the large format display 102 c. Due to theever-changing nature of computers and networks, the description ofcomputer system 110 depicted in FIG. 8 is intended only as a specificexample for purposes of illustrating the preferred embodiments of thepresent invention. Many other configurations of the computer system arepossible having more or less components than the computer systemdepicted in FIG. 8. The same components and variations can also make upeach of the other devices 102 in the collaboration environment of FIG.1, as well as the collaboration server 105 and display database 106.

Certain information about the drawing regions active on the digitaldisplay 102 c are stored in a database accessible to the computer system110 of the display client. The database can take on many forms indifferent embodiments, including but not limited to a MongoDB database,an XML database, a relational database, or an object oriented database.FIG. 9 is a schematic diagram illustrating certain information that thedatabase contains, and certain relationships among the data.

In embodiments described herein, each drawing region is considered to bea child of a toolbar. The touching of a point on the wall backgroundspawns a toolbar, which in turn spawns a drawing region (though thetoolbar is not necessarily visible until the drawing region opens).Similarly, to close a drawing region, a user touches a “close” icon onthe drawing region's toolbar. Thus in FIG. 9, the database is headed byone or more toolbar IDs 1110. Each toolbar ID 1110 includes or points toa respective block 1112 of data, indicating the horizontal position ofthe toolbar, the horizontal position of the left edge of the toolbar'sdrawing region, with width of the drawing region, and a set of drawingproperties for the drawing region. It will be appreciated that manyvariations are possible, such as specifying the right edge position ofthe drawing region rather than the left, and specifying the oppositeedge position rather than the drawing region width. The toolbar positionhas only a horizontal value, because in an embodiment, it always remainsat the same vertical position. In another embodiment both horizontal andvertical positions may be specified.

The drawing properties include or point to an array 1114 of drawingattributes, each in association with one or more values. The drawingproperties in FIG. 9 include a brush type, the value of which may forexample indicate “paint,” “ink,” “crayon,” “marker” or “eraser,” each ofwhich has a different character of appearance when drawn on the display102 c. The drawing properties in FIG. 9 also include a brush width,which can take on any value in a range of available values. The drawingproperties in FIG. 9 also include a brush color, which has threeassociated values: red, green and blue content. As used herein, thethree attributes brush type, brush width and brush color are consideredto constitute “line appearance properties.” Drawing attributes 1114 mayin various embodiments also include other attributes, such as those thataffect the location of the line or the location of part of the line.These properties may include such attributes as corner-rounding radius,or Bézier curve parameters. As can be seen in FIG. 11, there is norequirement that the drawing attributes (including the line appearanceproperties) for different drawing regions be the same. They can beestablished independently of each other, so there is no need that theybe identical. In a typical case they will not be identical.

In order to draw a line on the display 102 c, a user provides “drawinguser input” which indicates the drawing of the line. While otherembodiments may allow a user to draw with a finger, in the embodiment ofFIG. 1, only a stylus can be used to indicate the drawing of a line.Intuitively, the user so indicates by touching the stylus to the display102 c surface, within a drawing region, and dragging the stylus alongthe positions desired for the line. The end of a line drawing operationis indicated by lifting the stylus from the display 102 c surface. Thelocal computer system 110 determines from the user input where thepoints on the line are to be positioned, and displays them on thedisplay 102 c. The computer system 110 also transmits the strokeinformation to the collaboration server 105 (FIG. 1B), which writes theinformation into its display database 106 and transmits it back to thevarious devices 102 sharing the session. Each of the devices 102 canthen display the line (so long as the line intersects the device'sviewport), so all such devices 102 will show the line at roughly thesame time.

FIGS. 10-14 are flowcharts illustrating logic executed by the server,the display clients, or both. The logic can be implemented usingprocessors programmed using computer programs stored in memoryaccessible to the computer systems and executable by the processors, bydedicated logic hardware, including field programmable integratedcircuits, and by combinations of dedicated logic hardware and computerprograms. As with all flowcharts herein, it will be appreciated thatmany of the steps can be combined, performed in parallel or performed ina different sequence without affecting the functions achieved. In somecases, as the reader will appreciate, a re-arrangement of steps willachieve the same results only if certain other changes are made as well.In other cases, as the reader will appreciate, a re-arrangement of stepswill achieve the same results only if certain conditions are satisfied.Furthermore, it will be appreciated that the flow charts herein showonly steps that are pertinent to an understanding of the invention, andit will be understood that numerous additional steps for accomplishingother functions can be performed before, after and between those shown.

FIG. 10 illustrates basic logic executed on the server-side when a userjoins a session as part of a persistent workspace. The flowchart beginswith a login by the user (1210), in which the user may enter a useridentifier in a web portal access through a device possessed by theuser, such as a personal computer, a touchpad, a smart phone, etc. Next,a user authentication protocol is executed (1212), where a protocol, forexample, can include requiring the user to enter a personal password, toverify that the user is in fact a person who has entered the useridentifier. Next, the collaboration server, using for example the portalmachine, can present links to workspaces in which the authenticated useris authorized to participate (1214). Next, the collaboration server candetermine a selected display client, and a selected workspace for theuser (1216). This determination can be made by an exchange of messagesbetween the user possessed machine, and the portal using thecommunication channel on which the authentication protocol is executed.When the display client and workspace are identified, the collaborationserver can enable the display client to download data for the selectedworkspace (1218). Also, the collaboration server can add the client to alive event channel for the selected workspace (1220).

FIG. 11 illustrates basic two-channel logic executed on the client-sidewhen a user joins a workspace. The flowchart begins with a login by theuser (1230) executed on a first channel, in which the user may enter auser identifier and transmit it to the web portal. Next, the userauthentication protocol is executed (1232). The user then opens a pageat the portal using the communication channel on which theauthentication protocol is executed, which displays links for authorizedworkspaces (1234). Next, the user enters data identifying a selectedworkspace and display client to be used in the current session (1236)using the first channel. After the server enables the selected displayclient, the user activity can transfer to a channel between the displayclient and the server, which can then download the workspace data forthe selected session (1238). The display client can then traverse theworkspace data and construct an image for the display area managed bythe display client (1240). Also, the display client can then join thelive event channel (1242). The client-side network node and theserver-side network node can establish a protocol for encryption anddecryption of the spatial event map data during establishment of thesession.

In one example, the process of downloading the workspace data includesdelivering the event objects for the session to each display client.Included with the workspace data, a current user location can beprovided. Alternatively, the workspace data can be delivered, followedby a sequence of messages which identify to the display client how tocompute an offset from a default location, such as at the center of theworkspace data, to a current location associated with the user. Eachdisplay client then can traverse the event objects to identify thoseobjects having session locations which map to the display area managedby the display client. The logic to traverse the event objects caninclude an R-TREE search for example, which is configured to findobjects in the workspace that map to the display area. The identifiedobjects can then be rendered, possibly communicating with the portal toobtain data relevant to the objects, on the display area managed by thedisplay.

FIG. 12 illustrates basic logic on the client-side related todownloading workspace data. Logic begins with a download of workspacedata from the collaboration server (1250). The display client rendersthe objects that are within the range of the display client around theuser focus (1252), where the user focus is determined from the locationwithin the workspace that can be provided by the server, or maintainedat the client-side network node. The display client holds the workspacedata, or at least portions of the workspace data, including objectshaving current locations in workspace data, that are close to the userfocus. During the session, in response to user input or other data, thedisplay client traverses locations in the workspace to determine thecurrent location in the workspace (1254). The display client thenrenders objects within the range around the traversed locations (1256).At the end of the session, the last location within the workspace mappedby the display client is saved as the user location on a collaborationserver (1258).

FIG. 13 illustrates logic executed by the server-side for managingaccess to displays to which access is shared among many users. In thisexample, the server maintains a list of free display walls having activelinks to the server. These links can be set up when the display wallsare turned on, and maintained during waiting periods in which thedisplay wall is idle. The server can include logic to determine whethera free display wall has a link to the server (1260). If a wall isdetected that has a link to the server, then it is assigned a one-timeidentification code or PIN. When the identification code is assigned,then the wall is added to a “lobby,” which comprises a list of availablewalls (i.e. free walls) in the collaboration system (1264). The serveralso executes a loop waiting for authorized user logins (1266). When alogin is detected, the server prompts the user to select a workspace andto select a wall, for which the user is authorized (1268). The serverthen requires the user to enter the one-time identification codeassociated with the selected wall (1270). If the server does not receivea matching identification code for the selected wall (1272), then anerror signal is issued (1273). When the server receives a matchingidentification code for the selected wall (1272), then the displayclient or clients associated with the selected wall are linked to a liveevent channel for the session, and the one-time identification code isdisabled or changed, while the wall is occupied (1274). Also, the serversends the workspace data to the selected display client or clients(1276). The user is then able to collaborate with the session afterreceiving the workspace data (1278). When the user logs off of thesession, then the display wall is freed (1280). If the display wallremains available, it can be indicated to be a free display wall to theserver, and added to the lobby with a new identification code, followingsequence of steps 1260, 1262, 1264. In some embodiments, theidentification code is changed upon expiration of a time out interval,providing security against logins by intruders who might steal theidentification code from a wall that is not in use.

FIG. 14 illustrates basic logic executed on the server-side to manage afederated display array. The first step in this flowchart involvesdownloading the workspace data to each of the display clients in thearray (1302). Each display client renders objects within the range ofthe display client around a client offset from the user location (1304).The server monitors for client event messages (1306). When the serverreceives a client event message from one of the display clients in thearray, it determines whether the message relates to the workspace dataor only the array (1308). Array messages are broadcast on an arraychannel so that only those display clients participating in thefederated display array receive the messages (1310). Workspace datamessages are broadcast on the collaboration channel, so that all of thedisplay clients participating in sessions with workspace data areupdated as appropriate (1312). Those messages that relate only to thefederated display array, can include such messages as those that updatethe location of toolbars and drawing regions as described above. Also,messages that do not change the location of objects in the workspace,and do not create or modify objects that are part of the workspace data,can be determined to be local array only messages.

FIG. 15 illustrates, in the style of FIG. 1B, a system supportingdistributed display collaboration where there are displays distributedwidely. The system includes a collaboration server 105, with anassociated display database 106 storing workspace data. Thecollaboration server is connected by communication links 104 to aplurality of walls 1502 a, 1502 b, 1502 c which might be located forexample in Chicago, Los Angeles and Sao Paulo. The collaboration server105 is also coupled to a user device 1504, such as a touchpad or otherpersonal computing platform, which can be expected to be in thepossession of a known user. As mentioned above in connection with FIG.13, the collaboration server 105 can maintain a list of free displaywalls in a data structure referred to as a “lobby” 109. Associated witheach of the display walls is a one-time identification code, includingOT-PIN#1 associated with the display wall in Chicago, OT-PIN#2associated with the display wall in Los Angeles, and OT-PIN#3 associatedwith the display wall in Sao Paulo. A user in possession of the personaldevice 1504 can login to the portal managed by the collaboration server105, entering a user ID and a user password for the purposes of userauthentication. Then, the user in possession of the personal device 1504can provide a workspace identifier and an identification code for adisplay wall to which the user wants workspace data to be displayed.When the collaboration server successfully authenticates the user, anddetermines that the user has identified a display wall for which theuser is authorized, and a workspace for which the user is authorized,the display client associated with the identified display can be linkedto the collaboration event channel and enabled to download workspacedata. When the display device is enabled for a given session, it isremoved from the lobby 109, and the one-time identification code isdeleted or changed. Each time the display is added to the lobby 109, anew one-time identification code can be computed to accept user inputthat can contribute to the workspace data. The system can includemanagement logic providing workspace data to selected displays based aprotocol that insures that a user authorized for the workspace data hasphysical access the selected display.

FIG. 16 is a simplified diagram of a client-side network node, includinga client processor 1600, a display driver 1601, a local display and userinterface such as a touchscreen 1602, a protocol stack 1604 including acommunication interface controlled by the stack, local memory 1605storing a cache copy of the live spatial event map and a cache of imagesand other graphical constructs used in rendering the displayable area,and input protocol device 1607 which executes a input protocol whichtranslates input from a tangible user input device such as atouchscreen, or a mouse, into a form usable by a command interpreter1606. A suitable input protocol device 1607 can include softwarecompatible with a TUIO industry-standard, for example for interpretationof tangible and multi-touch interaction with the display wall. Theprotocol stack 1604 receives API compliant messages and Internetmessages from the client processor 1600 and as discussed above includesresources to establish a channel 1611 to a collaboration server acrosswhich API compliant messages can be exchanged, and a link 1610 to theInternet in support of other communications that serve the local display1602. The display driver 1601 controls a displayable area 1603 on thelocal display 1602. The displayable area 1603 can be logicallyconfigured by the client processor or other programming resources in theclient-side network node. Also, the physical size of the displayablearea 1603 can be fixed for a given implementation of the local display.The client processor 1600 can include processing resources such as abrowser, mapping logic used for translating between locations on thedisplayable area 1603 and the workspace, and logic to implement APIprocedures.

A system for collaboration described herein can include a client-sidenetwork node like that of FIG. 16. The client-side network node includesa display having a physical display space, a user input device, aprocessor and a communication port. The client-side network node isconfigured with logic:

to establish a link to a server-side network node;

to retrieve from the server-side network node at least part of a spatialevent log of events relating to graphical targets having locations in aworkspace, entries in the log including a location in the workspace ofthe graphical target of an event, a time of the event, and a targetidentifier of the graphical target;

to map a displayable area in the physical display space to a mapped areawithin the workspace, to identify entries in the spatial event logwithin the mapped area, render graphical targets identified by theidentified entries onto the displayable area;

to accept input data from the user input device creating events relatingto modification and creation of graphical targets displayed within thedisplayable area, and to send messages based upon the events to theserver-side network node.

The client-side network node shown in FIG. 16 illustrates an exampleincluding an application interface including a process to communicatewith the server-side network node.

The client-side network node shown in FIG. 16 illustrates an exampleconfigured according to an API, wherein the events include a first classof event designated as history events to be distributed among otherclient-side network nodes and to be added to the spatial event log inthe server-side network node, and a second class of event designated asephemeral to be distributed among other client-side network nodes butnot added to the spatial event log in the server-side network node.

FIG. 17 is a simplified flow diagram of a procedure executed by theclient-side network node. The order illustrated in the simplified flowdiagram is provided for the purposes of illustration, and can bemodified as suits a particular implementation. Many of the steps forexample, can be executed in parallel. In this procedure, a client loginis executed (1700) by which the client is given access to a specificcollaboration session and its spatial event map. The collaborationserver provides an identifier of, or identifiers of parts of, thespatial event map which can be used by the client to retrieve thespatial event map from the collaboration server (1701). The clientretrieves the spatial event map, or at least portions of it, from thecollaboration server using the identifier or identifiers provided(1702).

For example, the client can request all history for a given workspace towhich it has been granted access as follows:

curl http://localhost:4545/<sessionId>/history

The server will respond with all chunks (each its own section of time):

[“/<sessionId>/history/<startTime>/<endTime>?b=1”][“/<sessionId>/history/<startTime>/<endTime>?b=1”]

For each chunk, the client will request the events:

Curlhttp://localhost:4545/<sessionId>/history/<startTime>/<endTime>?b=<cache-buster>

Each responded chunk is an array of events and is cacheable by theclient:

[ [ 4, ″sx″, ″4.4″, [537, 650, 536, 649, 536, 648, ...], { “size″: 10,″color″: [0, 0, 0, 1], ″brush″: 1  }, 1347644106241, ″cardFling″ ] ]

The individual messages might include information like position onscreen, color, width of stroke, time created etc.

The client then determines a location in the workspace, using forexample a server provided focus point, and display boundaries for thelocal display (1703). The local copy of the spatial event map istraversed to gather display data for spatial event map entries that mapto the displayable area for the local display. In some embodiments, theclient may gather additional data in support of rendering a display forspatial event map entries within a culling boundary defining a regionlarger than the displayable area for the local display, in order toprepare for supporting predicted user interactions such as zoom and panwithin the workspace (1704). The client processor executes a processusing spatial event map events, ephemeral events and display data torender parts of the spatial event map that fall within the displayboundary (1705). This process receives local user interface messages,such as from the TUIO driver (1706). Also, this process receives socketAPI messages from the collaboration server (1710). In response to localuser interface messages, the process can classify inputs as historyevents and ephemeral events, send API messages on the socket to thecollaboration server for both history events and ephemeral events asspecified by the API, update the cached portions of the spatial eventmap with history events, and produce display data for both historyevents and ephemeral events (1707). In response to the socket APImessages, the process updates the cached portion of the spatial eventmap with history events identified by the server-side network node,responds to API messages on the socket as specified by the API, andproduce display data for both history events and ephemeral events aboutwhich it is notified by the socket messages (1711).

FIG. 18 is a simplified flow diagram of a process for interpreting userinput executed by a client-side network node. The order illustrated inthe simplified flow diagram is provided for the purposes ofillustration, and can be modified as suits a particular implementation.Many of the steps for example, can be executed in parallel. The processbegins with receiving a native I/O event type message from a tangibleuser input device, along with a physical coordinate on the display(1800). The process maps the physical coordinate on the display to acoordinate in the workspace and identifies using its local cached copyof the spatial event map, objects in the spatial event map which map tothe workspace coordinate, if any (1801). Then, based on the workspacecoordinate, the identified object, context and I/O event type, theprocessor determines an API event type, including whether the event is ahistory event or ephemeral event (1802). The client-side processor thenproduces an API compliant message, updates the local cached copy of thespatial event map, and updates the display data (1803). Using aserver-side network node and a client-side network node as describedhere, some basic procedures include logging in and downloading a spatialevent map for a session to a client-side network node, and connecting toa workspace channel of live workspace spatial events.

Logging in and Downloading Spatial Event Map.

1. The client request authorization to join a collaboration session, andopen a workspace.

2. The server authorizes the client to participate in the session, andbegin loading the spatial event map for the workspace.

3. The client requests an identification, such as a “table of contents”of the spatial event map associated with the session.

4. Each portion of the spatial event map identified in the table ofcontents is requested by the client. These portions of the spatial eventmap together represent the workspace as a linear sequence of events fromthe beginning of workspace-time to the present. The “beginning ofworkspace-time” can be considered an elapsed time from the time ofinitiation of the collaboration session, or an absolute time recorded inassociation with the session.

5. The client assembles a cached copy of the spatial event map in itslocal memory.

6. The client displays an appropriate region of the workspace using itsspatial event map to determine what is relevant given the currentdisplayable area or viewport on the local display.

Connecting to the session channel of live spatial event map events:

1. After authorization, a client requests to join a workspace channel.

2. The server adds the client to the list of workspace participants toreceive updates via the workspace channels.

3. The client receives live messages from the workspace that carry bothhistory events and ephemeral events, and a communication paradigm like achat room. For example, a sequence of ephemeral events, and a historyevent can be associated with moving object in the spatial event map to anew location in the spatial event map.

4. The client reacts to live messages from the server-side network nodeby altering its local copy of the spatial event map and re-rendering itslocal display.

5. Live messages consist of “history” events which are to be persistedas undue-double, recorded events in the spatial event map, and“ephemeral” events which are pieces of information that do not becomepart of the history of the session.

6. When a client creates, modifies, moves or deletes an object byinteraction with its local display, a new event is created by theclient-side network node and sent across the workspace channel to theserver-side network node. The server-side network node saves historyevents in the spatial event map for the session, and distributes bothhistory events and ephemeral events to all active clients in thesession.

7. When exiting the session, the client disconnects from the workspacechannel.

A collaboration system can have many, distributed digital displays whichare used both to display images based on workspace data managed by ashared collaboration server, and to accept user input that cancontribute to the workspace data, while enabling each display to rapidlyconstruct an image to display based on session history, real time localinput and real-time input from other displays.

As used herein, the “identification” of an item of information does notnecessarily require the direct specification of that item ofinformation. Information can be “identified” in a field by simplyreferring to the actual information through one or more layers ofindirection, or by identifying one or more items of differentinformation which are together sufficient to determine the actual itemof information. In addition, the term “indicate” is used herein to meanthe same as “identify”.

Also as used herein, a given signal, event or value is “responsive” to apredecessor signal, event or value if the predecessor signal, event orvalue influenced the given signal, event or value. If there is anintervening processing element, step or time period, the given signal,event or value can still be “responsive” to the predecessor signal,event or value. If the intervening processing element or step combinesmore than one signal, event or value, the signal output of theprocessing element or step is considered “responsive” to each of thesignal, event or value inputs. If the given signal, event or value isthe same as the predecessor signal, event or value, this is merely adegenerate case in which the given signal, event or value is stillconsidered to be “responsive” to the predecessor signal, event or value.“Dependency” of a given signal, event or value upon another signal,event or value is defined similarly.

The applicant hereby discloses in isolation each individual featuredescribed herein and any combination of two or more such features, tothe extent that such features or combinations are capable of beingcarried out based on the present specification as a whole in light ofthe common general knowledge of a person skilled in the art,irrespective of whether such features or combinations of features solveany problems disclosed herein, and without limitation to the scope ofthe claims. The applicant indicates that aspects of the presentinvention may consist of any such feature or combination of features. Inview of the foregoing description it will be evident to a person skilledin the art that various modifications may be made within the scope ofthe invention.

The foregoing description of preferred embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in this art.For example, though the displays described herein are of large format,small format displays can also be arranged to use multiple drawingregions, though multiple drawing regions are more useful for displaysthat are at least as large as 12 feet in width. In particular, andwithout limitation, any and all variations described, suggested by theBackground section of this patent application or by the materialincorporated by reference are specifically incorporated by referenceinto the description herein of embodiments of the invention. Inaddition, any and all variations described, suggested or incorporated byreference herein with respect to any one embodiment are also to beconsidered taught with respect to all other embodiments. The embodimentsdescribed herein were chosen and described in order to best explain theprinciples of the invention and its practical application, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with various modifications as are suited to theparticular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. A system comprising: a client-side network nodeincluding a display having a physical display space, a user inputdevice, a processor and a communication port, the client-side networknode being configured with logic: to establish a link to a server-sidenetwork node; to retrieve, from the server-side network node, at leastpart of a log of entries to identify events in a workspace, theworkspace comprising locations having virtual coordinates, the eventsidentified by the entries in the log being related to graphical targetshaving virtual coordinates within the workspace, wherein an entry in thelog, which identifies an event, comprises data specifying virtualcoordinates of a location within the workspace of the graphical targetrelated to the event, a target identifier of the graphical targetrelated to the event to be displayed when rendered on the display of theclient-side network node, data identifying an action selected from agroup of actions including creation, movement and deletion of thegraphical target within the workspace, and a time of the event; to map adisplayable area in the physical display space to a mapped area withinthe workspace; to identify events in the retrieved log having locationswithin the mapped area; to render graphical targets identified by theidentified events onto the displayable area, wherein the retrieved logincludes at least one entry for an event that identifies creationaction, and at least one entry for an event that identifies a movementaction; to accept input data from the user input device creating eventsrelating to movement and creation of graphical targets displayed withinthe displayable area; and to send messages based upon the created eventsto the server-side network node.
 2. The system of claim 1, the logicincluding an application interface including a process to communicatewith the server-side network node.
 3. The system of claim 1, wherein themessages based on the created events include a first class of eventdesignated as history events to be distributed among other client-sidenetwork nodes and to be added to the log in the server-side networknode, and a second class of event designated as ephemeral events to bedistributed among other client-side network nodes but not added to thelog in the server-side network node.
 4. The system of claim 1, whereinthe graphical targets include text and strokes.
 5. The system of claim1, wherein the entries in the log include a parameter identifyinggraphical constructs used to render the graphical target on a display.6. The system of claim 1, wherein the entries in the log include aparameter identifying a user associated with an event.
 7. The system ofclaim 1, wherein the logic includes procedures to pan and zoom thedisplayable area relative to the workspace.
 8. The system of claim 1,wherein the workspace encompasses an unbounded virtual area.
 9. A methodcomprising: establishing a link from a client-side network node to aserver-side network node; retrieving, at the client-side network nodeand from the server-side network node, at least part of a log of entriesto identify events in a workspace, the workspace comprising locationshaving virtual coordinates, the events identified by the entries in thelog being related to graphical targets having virtual coordinates withinthe workspace, wherein an entry in the log, which identifies an event,comprises data specifying virtual coordinates of a location within theworkspace of the graphical target related to the event, a targetidentifier of the graphical target related to the event to be displayedwhen rendered on a display of the client-side network node, dataidentifying an action selected from a group of actions includingcreation, movement and deletion of the graphical target within theworkspace, and a time of the event; mapping a displayable area inphysical display space at the client-side network node to a mapped areawithin the workspace, to identify events in the retrieved log havinglocations within the mapped area, and to render graphical targetsidentified by the identified events onto the displayable area, whereinthe retrieved log includes at least one entry for an event thatidentifies creation action, and at least one entry for an event thatidentifies a movement action; accepting input data from a user inputdevice at the client-side network node creating events relating tomovement and creation of graphical targets displayed within thedisplayable area; and sending messages based upon the created events onthe established link to the server-side network node.
 10. The method ofclaim 9, including sending messages compliant with an applicationprogram interface executed at the server-side network node.
 11. Themethod of claim 9, wherein the messages based on the created eventsinclude a first class of event designated as history events to bedistributed among other client-side network nodes and to be added asentries to the log in the server-side network node, and a second classof event designated as ephemeral events to be distributed among otherclient-side network nodes but not added to the log in the server-sidenetwork node.
 12. The method of claim 9, wherein the graphical targetsinclude text and strokes.
 13. The method of claim 9, wherein the entriesin the log include a parameter identifying graphical constructs used torender the graphical target on a display.
 14. The method of claim 9,wherein the entries in the log include a parameter identifying a userassociated with an event.
 15. The method of claim 9, wherein the methodfurther includes executing procedures to pan and zoom the displayablearea relative to the workspace at the client-side network node.
 16. Themethod of claim 9, wherein the workspace encompasses an unboundedvirtual area.
 17. A product comprising: non-transitory computer readablememory; executable instructions stored in the memory, for establishing alink from a client-side network node to a server-side network node;executable instructions for retrieving, at the client-side network nodeand from the server-side network node, at least part of a log of entriesto identify events in a workspace, the workspace comprising locationshaving virtual coordinates, the events identified by the entries in thelog being related to graphical targets having virtual coordinates withinthe workspace, wherein an entry in the log, which identifies an event,comprises data specifying virtual coordinates of a location within theworkspace of the graphical target related to the event, a targetidentifier of the graphical target related to the event to be displayedwhen rendered on a display of the client-side network node, dataidentifying an action selected from a group of actions includingcreation, movement and deletion of the graphical target within theworkspace, and a time of the event; executable instructions for mappinga displayable area in a physical display space at the client-sidenetwork node to a mapped area within the workspace, to identify eventsin the retrieved log having locations within the mapped area, and torender graphical targets identified by the identified events onto thedisplayable area, wherein the retrieved log includes at least one entryfor an event that identifies creation action, and at least one entry foran event that identifies a movement action; executable instructions foraccepting input data from a user input device at the client-side networknode creating events relating to movement and creation of graphicaltargets displayed within the displayable area; and executableinstructions for sending messages based upon the created events on theestablished link to the server-side network node.
 18. The product ofclaim 17, including executable instructions for sending messagescompliant with an application program interface executed at theserver-side network node.
 19. The product of claim 17, wherein theevents include a first class of event designated as history events to bedistributed among other client-side network nodes and to be added to thelog in the server-side network node, and a second class of eventdesignated as ephemeral events to be distributed among other client-sidenetwork nodes but not added to the log in the server-side network node.20. The product of claim 17, wherein the graphical targets include textand strokes.
 21. The product of claim 17, wherein the entries in the loginclude a parameter identifying graphical constructs used to render thegraphical target on a display.
 22. The product of claim 17, wherein theentries in the log include a parameter identifying a user associatedwith an event.
 23. The product of claim 17, including executableinstructions for executing procedures to pan and zoom the displayablearea relative to the workspace at the client-side network node.
 24. Theproduct of claim 17, wherein the workspace encompasses an unboundedvirtual area.